Fix GGUF user-defined token atomicity - #315

Summary

Fix GGUF USER_DEFINED tokens losing their atomic tokenization behavior after conversion to a Hugging Face tokenizer.

Qwen3.6 GGUF contains and as USER_DEFINED (token_type = 4) vocabulary entries with fixed IDs. The IDs were preserved by the GGUF conversion, but encoding the token text could still split them into ordinary subword tokens.

For example, before this fix:

GGUF / llama.cpp:
-> [248068]
-> [248069]

FreeToken:
-> [13314, 741, 29]
-> [510, 26003, 29]

The rendered chat prompt itself was identical between FreeToken and llama.cpp; the difference was in tokenization.

Fix

After GGUF tokenizer conversion:

inspect tokenizer.ggml.token_type

restore only USER_DEFINED (type = 4) tokens that are not already atomic

register them as AddedToken(..., normalized=False, special=False)

preserve their existing vocabulary IDs

assert that vocabulary size does not change

special=False is intentional: GGUF USER_DEFINED tokens are distinct from GGUF CONTROL tokens.

A synthetic regression test reproduces the failure mode without requiring a model or GPU.

Validation

Tokenization

After the fix:

-> [248068]
-> [248069]

FreeToken and llama.cpp produce the same token sequence for the tested Qwen3.6 chat prompt.

End-to-end Qwen3.6 reasoning

Qwen3.6-35B-A3B Q4_K, same prompt/config:

before: 0/10 completed
after: 10/10 completed

Qwen3.6-35B-A3B IQ3_S:

post-fix case reproduction: 10/10 completed

Broader 47-case IQ3_S default-reasoning regression battery:

before: 20/47 completed (42.6%)
after: 42/47 completed (89.4%)

The remaining failures include long-context / reasoning-completion cases and are not being attributed to this tokenizer issue.

Tests

On the current upstream main plus this commit:

pytest tests/models/test_gguf_tokenizer.py tests/tokenizer/ -m "not slow"

28 passed

The regression test was also verified red/green: with the previous loader it fragmented the synthetic token; with this change it encodes to its declared GGUF vocabulary ID.

Scope

This change only affects GGUF tokens explicitly marked USER_DEFINED whose vocabulary ID already exists but whose text is not encoded atomically. It does not create new vocabulary entries or convert them into Hugging Face special tokens.